Skip to content

docs(observability): list agentest under evaluation - #9483

Open
Houyun Sun (sunhouy) wants to merge 2 commits into
crewAIInc:mainfrom
sunhouy:docs/observability-agentest
Open

Houyun Sun (sunhouy) wants to merge 2 commits into
crewAIInc:mainfrom
sunhouy:docs/observability-agentest

Conversation

@sunhouy

Copy link
Copy Markdown

Related issue

Fixes #9479

Summary

Adds agentest to the Observability section under Evaluation & Quality Assurance, next to the Patronus page. Three files: the new page, one card on the overview, one nav entry in docs/docs.json.

The page is short on purpose. agentest's API is still moving, so instead of pasting a full integration guide I describe what the checks assert on (tool calls and their arguments, output shape, guardrail state, step order), then a minimal pytest example and a note on how it relates to the tracing pages in the same section: tracing tells you what a run did, agentest turns that into something that fails on the next commit.

The example uses ScrapeWebsiteTool and asserts on its name, because that's the cheapest way to show a tool-level assertion without needing an LLM.

Verification

  • Tests added or updated for the changed behavior
  • Relevant tests and quality checks pass locally

Docs only, so no tests. What I did run:

  • The Python example on the page, against lib/crewai + lib/crewai-tools at 83d7407: returns ['Read website content'], which is what the assertion expects.
  • docs/docs.json parses as JSON after the edit; re-serialising the upstream blob with json.dumps(indent=2) gives byte-identical output, so the diff is one added line (checked with difflib against the blob sha).
  • overview.mdx diff is the card only; <CardGroup> nesting is unchanged.
  • Checked that no other open PR mentions agentest (search/issues?q=repo:crewAIInc/crewAI+agentest returns only [FEATURE] Docs: add agentest to observability evaluation listings #9479).

Additional context

I did not add ar/ko/pt-BR copies. The observability pages in those locales are maintained by the docs team during the [docs-freeze] snapshots, and the page links to the project README for anything version-specific. Happy to add translations if you'd rather have them in the same PR.

The reporter asked for the llm-generated label on #9479; this PR was written by hand.

Adds a page for agentest next to the Patronus evaluation page, plus a card
on the observability overview and a nav entry in docs/docs.json.

agentest runs deterministic checks over an agent (tool calls and their
arguments, output shape, guardrail state) without calling a model, so it
sits beside the tracing tools in this section rather than replacing them.
The page keeps the setup short and points at the project README for the
API, since that moves faster than a docs page can.

Refs crewAIInc#9479
@mintlify

mintlify Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Preview approval required

This pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit 5a97010.

Approve preview

@strix-security

strix-security Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Strix Security Review

Warning

This pull request has 1 commit after the last Strix review (752c5ae). Strix has not reviewed these changes.
Automatic review on push is off for this repository. To review the latest changes, tag @strix-security in a comment, or turn on re-review on push.

✅ No open security findings.

PR summary: Adds a documentation page for agentest under the Observability → Evaluation section, plus a card on the observability overview and a nav entry in docs/docs.json.

Review summary

Docs-only change: a new observability page, one card on the overview, and one nav entry in docs/docs.json. No application code, no input handling, no authentication or query paths are touched, and the JSON still parses cleanly. Nothing in the added MDX is user-controlled — it is static prose compiled at build time — so there is no injection, XSS or deserialization surface here.

One suspicion was investigated and rejected: the page instructs pip install agentest, while the project it links publishes under a different distribution name and the PyPI agentest name resolves to an unrelated publisher's library. The relevant wheel was inspected and is a benign, unrelated package with no install-time hooks, telemetry or credential access, and nothing in the reviewed change consumes it. The defect is a wrong distribution name in prose — a docs-accuracy problem worth correcting so readers install the tool the page documents, not a vulnerability in the changed code.

Ruled out (1)
  • Does the new page's `pip install agentest` instruction expose CrewAI users to an unrelated or malicious third-party package?
    The PyPI `agentest` name resolves to a benign unrelated library — its wheel has no install-time hooks, telemetry, credential access or exec/eval of external content — and it is merely named in prose, so no attacker-controlled input reaches any CrewAI code path; the mismatch is a documentation accur…

Review 1 on this PR · 752c5ae · requested Nov 16 07:33 UTC when the PR was opened


Reviewed by Strix
Re-run review · Configure security review settings · Try a deeper review with @strix-security /deep

Comment thread docs/edge/en/observability/agentest.mdx Resolved
<Steps>
<Step title="Install">
```shell
pip install "git+https://github.com/srpadmanabhuni/agentest"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Agentic Security Review

Severity: CRITICAL

Description: The new observability page tells readers to pip install "git+https://github.com/srpadmanabhuni/agentest" with no commit SHA, tag, or locked revision. That URL tracks the default branch main of a personal GitHub repo CrewAI does not own. pip builds whatever is on that branch at install time, and the same page tells readers to run the resulting checks beside their tests and in CI.

Impact: Anyone who can push to github.com/srpadmanabhuni/agentest, or who later takes over that account, can change the code that official CrewAI docs instruct customers and CI to fetch and execute. A later push needs no further docs edit. Install runs the repo build backend as the installing user, and the page sends follow-on test imports and CI runs at that same moving tree, so the follower's source and environment credentials are in the blast radius. Sibling observability pages install named PyPI distributions; this is the only git+https:// installer in the docs. The repo's own README already uses a tag pin (@v3.3.2), while main's pyproject is already 3.4.0, so the documented command is not a reviewed snapshot.

Remediation: Remove the floating VCS install from the published page until a reviewed artifact exists. A full 40-character commit SHA that CrewAI has reviewed is an acceptable interim pin if a GitHub install must stay; a git tag is not, because tags move. Prefer a versioned release under a controlled package name once it exists. Keep the warning not to install the unrelated PyPI project named agentest.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE] Docs: add agentest to observability evaluation listings

1 participant